<!-- HTML header for doxygen 1.8.6-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.13"/>
<title>OpenCV: Conversion of TensorFlow Detection Models and Launch with OpenCV Python</title>
<link href="../../opencv.ico" rel="shortcut icon" type="image/x-icon" />
<link href="../../tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../jquery.js"></script>
<script type="text/javascript" src="../../dynsections.js"></script>
<script type="text/javascript" src="../../tutorial-utils.js"></script>
<link href="../../search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../search/searchdata.js"></script>
<script type="text/javascript" src="../../search/search.js"></script>
<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
    jax: ["input/TeX","output/HTML-CSS"],
});
//<![CDATA[
MathJax.Hub.Config(
{
  TeX: {
      Macros: {
          matTT: [ "\\[ \\left|\\begin{array}{ccc} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{array}\\right| \\]", 9],
          fork: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ \\end{array} \\right.", 4],
          forkthree: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ \\end{array} \\right.", 6],
          forkfour: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ #7 & \\mbox{#8}\\\\ \\end{array} \\right.", 8],
          vecthree: ["\\begin{bmatrix} #1\\\\ #2\\\\ #3 \\end{bmatrix}", 3],
          vecthreethree: ["\\begin{bmatrix} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{bmatrix}", 9],
          cameramatrix: ["#1 = \\begin{bmatrix} f_x & 0 & c_x\\\\ 0 & f_y & c_y\\\\ 0 & 0 & 1 \\end{bmatrix}", 1],
          distcoeffs: ["(k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6 [, s_1, s_2, s_3, s_4[, \\tau_x, \\tau_y]]]]) \\text{ of 4, 5, 8, 12 or 14 elements}"],
          distcoeffsfisheye: ["(k_1, k_2, k_3, k_4)"],
          hdotsfor: ["\\dots", 1],
          mathbbm: ["\\mathbb{#1}", 1],
          bordermatrix: ["\\matrix{#1}", 1]
      }
  }
}
);
//]]>
</script><script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js"></script>
<link href="../../doxygen.css" rel="stylesheet" type="text/css" />
<link href="../../stylesheet.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<!--#include virtual="/google-search.html"-->
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr style="height: 56px;">
  <td id="projectlogo"><img alt="Logo" src="../../opencv-logo-small.png"/></td>
  <td style="padding-left: 0.5em;">
   <div id="projectname">OpenCV
   &#160;<span id="projectnumber">4.5.2</span>
   </div>
   <div id="projectbrief">Open Source Computer Vision</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.13 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "../../search",false,'Search');
</script>
<script type="text/javascript" src="../../menudata.js"></script>
<script type="text/javascript" src="../../menu.js"></script>
<script type="text/javascript">
$(function() {
  initMenu('../../',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
</script>
<div id="main-nav"></div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

<div id="nav-path" class="navpath">
  <ul>
<li class="navelem"><a class="el" href="../../d9/df8/tutorial_root.html">OpenCV Tutorials</a></li><li class="navelem"><a class="el" href="../../d2/d58/tutorial_table_of_content_dnn.html">Deep Neural Networks (dnn module)</a></li>  </ul>
</div>
</div><!-- top -->
<div class="header">
  <div class="headertitle">
<div class="title">Conversion of TensorFlow Detection Models and Launch with OpenCV Python </div>  </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><table class="doxtable">
<tr>
<th align="right"></th><th align="left"></th></tr>
<tr>
<td align="right">Original author </td><td align="left">Anastasia Murzova </td></tr>
<tr>
<td align="right">Compatibility </td><td align="left">OpenCV &gt;= 4.5 </td></tr>
</table>
<h2>Goals</h2>
<p>In this tutorial you will learn how to:</p><ul>
<li>obtain frozen graphs of TensorFlow (TF) detection models</li>
<li>run converted TensorFlow model with OpenCV Python API</li>
</ul>
<p>We will explore the above-listed points by the example of SSD MobileNetV1.</p>
<h2>Introduction</h2>
<p>Let's briefly view the key concepts involved in the pipeline of TensorFlow models transition with OpenCV API. The initial step in the conversion of TensorFlow models into <a class="el" href="../../db/d30/classcv_1_1dnn_1_1Net.html" title="This class allows to create and manipulate comprehensive artificial neural networks. ">cv.dnn.Net</a> is obtaining the frozen TF model graph. A frozen graph defines the combination of the model graph structure with kept values of the required variables, for example, weights. The frozen graph is saved in <a href="https://en.wikipedia.org/wiki/Protocol_Buffers">protobuf</a> (<code>.pb</code>) files. There are special functions for reading <code>.pb</code> graphs in OpenCV: <a class="el" href="../../d6/d0f/group__dnn.html#gad820b280978d06773234ba6841e77e8d" title="Reads a network model stored in TensorFlow framework&#39;s format. ">cv.dnn.readNetFromTensorflow</a> and <a class="el" href="../../d6/d0f/group__dnn.html#ga3b34fe7a29494a6a4295c169a7d32422" title="Read deep learning network represented in one of the supported formats. ">cv.dnn.readNet</a>.</p>
<h2>Requirements</h2>
<p>To be able to experiment with the below code you will need to install a set of libraries. We will use a virtual environment with python3.7+ for this:</p>
<div class="fragment"><div class="line">virtualenv -p /usr/bin/python3.7 &lt;env_dir_path&gt;</div><div class="line">source &lt;env_dir_path&gt;/bin/activate</div></div><!-- fragment --><p>For OpenCV-Python building from source, follow the corresponding instructions from the <a class="el" href="../../da/df6/tutorial_py_table_of_contents_setup.html">Introduction to OpenCV</a>.</p>
<p>Before you start the installation of the libraries, you can customize the <a href="https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/requirements.txt">requirements.txt</a>, excluding or including (for example, <code>opencv-python</code>) some dependencies. The below line initiates requirements installation into the previously activated virtual environment:</p>
<div class="fragment"><div class="line">pip install -r requirements.txt</div></div><!-- fragment --><h2>Practice</h2>
<p>In this part we are going to cover the following points:</p><ol type="1">
<li>create a TF classification model conversion pipeline and provide the inference</li>
<li>provide the inference, process prediction results</li>
</ol>
<h3>Model Preparation</h3>
<p>The code in this subchapter is located in the <code>samples/dnn/dnn_model_runner</code> module and can be executed with the below line:</p>
<div class="fragment"><div class="line">python -m dnn_model_runner.dnn_conversion.tf.detection.py_to_py_ssd_mobilenet</div></div><!-- fragment --><p>The following code contains the steps of the TF SSD MobileNetV1 model retrieval:</p>
<div class="fragment"><div class="line">tf_model_name = &#39;ssd_mobilenet_v1_coco_2017_11_17&#39;</div><div class="line">graph_extraction_dir = &quot;./&quot;</div><div class="line">frozen_graph_path = extract_tf_frozen_graph(tf_model_name, graph_extraction_dir)</div><div class="line">print(&quot;Frozen graph path for {}: {}&quot;.format(tf_model_name, frozen_graph_path))</div></div><!-- fragment --><p>In <code>extract_tf_frozen_graph</code> function we extract the provided in model archive <code>frozen_inference_graph.pb</code> for its further processing:</p>
<div class="fragment"><div class="line"># define model archive name</div><div class="line">tf_model_tar = model_name + &#39;.tar.gz&#39;</div><div class="line"># define link to retrieve model archive</div><div class="line">model_link = DETECTION_MODELS_URL + tf_model_tar</div><div class="line"></div><div class="line">tf_frozen_graph_name = &#39;frozen_inference_graph&#39;</div><div class="line"></div><div class="line">try:</div><div class="line">    urllib.request.urlretrieve(model_link, tf_model_tar)</div><div class="line">except Exception:</div><div class="line">    print(&quot;TF {} was not retrieved: {}&quot;.format(model_name, model_link))</div><div class="line">    return</div><div class="line"></div><div class="line">print(&quot;TF {} was retrieved.&quot;.format(model_name))</div><div class="line"></div><div class="line">tf_model_tar = tarfile.open(tf_model_tar)</div><div class="line">frozen_graph_path = &quot;&quot;</div><div class="line"></div><div class="line">for model_tar_elem in tf_model_tar.getmembers():</div><div class="line">    if tf_frozen_graph_name in os.path.basename(model_tar_elem.name):</div><div class="line">        tf_model_tar.extract(model_tar_elem, extracted_model_path)</div><div class="line">        frozen_graph_path = os.path.join(extracted_model_path, model_tar_elem.name)</div><div class="line">        break</div><div class="line">tf_model_tar.close()</div></div><!-- fragment --><p>After the successful execution of the above code we will get the following output:</p>
<div class="fragment"><div class="line">TF ssd_mobilenet_v1_coco_2017_11_17 was retrieved.</div><div class="line">Frozen graph path for ssd_mobilenet_v1_coco_2017_11_17: ./ssd_mobilenet_v1_coco_2017_11_17/frozen_inference_graph.pb</div></div><!-- fragment --><p>To provide model inference we will use the below <a href="https://www.pexels.com/photo/bus-and-car-on-one-way-street-3626589/">double-decker bus photo</a> (under <a href="https://www.pexels.com/license/">Pexels</a> license):</p>
<div class="image">
<img src="../../pexels_double_decker_bus.jpg" alt="pexels_double_decker_bus.jpg"/>
<div class="caption">
Double-decker bus</div></div>
<p> To initiate the test process we need to provide an appropriate model configuration. We will use <a href="https://github.com/tensorflow/models/blob/master/research/object_detection/samples/configs/ssd_mobilenet_v1_coco.config"><code>ssd_mobilenet_v1_coco.config</code></a> from <a href="https://github.com/tensorflow/models/tree/master/research/object_detection#tensorflow-object-detection-api">TensorFlow Object Detection API</a>. TensorFlow Object Detection API framework contains helpful mechanisms for object detection model manipulations.</p>
<p>We will use this configuration to provide a text graph representation. To generate <code>.pbtxt</code> we will use the corresponding <a href="https://github.com/opencv/opencv/blob/master/samples/dnn/tf_text_graph_ssd.py"><code>samples/dnn/tf_text_graph_ssd.py</code></a> script:</p>
<div class="fragment"><div class="line">python tf_text_graph_ssd.py --input ssd_mobilenet_v1_coco_2017_11_17/frozen_inference_graph.pb --config ssd_mobilenet_v1_coco_2017_11_17/ssd_mobilenet_v1_coco.config --output ssd_mobilenet_v1_coco_2017_11_17.pbtxt</div></div><!-- fragment --><p>After successful execution <code>ssd_mobilenet_v1_coco_2017_11_17.pbtxt</code> will be created.</p>
<p>Before we run <code>object_detection.py</code>, let's have a look at the default values for the SSD MobileNetV1 test process configuration. They are located in <a href="https://github.com/opencv/opencv/blob/master/samples/dnn/models.yml"><code>models.yml</code></a>:</p>
<div class="fragment"><div class="line">ssd_tf:</div><div class="line">  model: &quot;ssd_mobilenet_v1_coco_2017_11_17.pb&quot;</div><div class="line">  config: &quot;ssd_mobilenet_v1_coco_2017_11_17.pbtxt&quot;</div><div class="line">  mean: [0, 0, 0]</div><div class="line">  scale: 1.0</div><div class="line">  width: 300</div><div class="line">  height: 300</div><div class="line">  rgb: true</div><div class="line">  classes: &quot;object_detection_classes_coco.txt&quot;</div><div class="line">  sample: &quot;object_detection&quot;</div></div><!-- fragment --><p>To fetch these values we need to provide frozen graph <code>ssd_mobilenet_v1_coco_2017_11_17.pb</code> model and text graph <code>ssd_mobilenet_v1_coco_2017_11_17.pbtxt</code>:</p>
<div class="fragment"><div class="line">python object_detection.py ssd_tf --input ../data/pexels_double_decker_bus.jpg</div></div><!-- fragment --><p>This line is equivalent to:</p>
<div class="fragment"><div class="line">python object_detection.py --model ssd_mobilenet_v1_coco_2017_11_17.pb --config  ssd_mobilenet_v1_coco_2017_11_17.pbtxt  --input ../data/pexels_double_decker_bus.jpg --width 300 --height 300 --classes ../data/dnn/object_detection_classes_coco.txt</div></div><!-- fragment --><p>The result is:</p>
<div class="image">
<img src="../../opencv_bus_res.jpg" alt="opencv_bus_res.jpg"/>
<div class="caption">
OpenCV SSD bus result</div></div>
<p> There are several helpful parameters, which can be also customized for result corrections: threshold (<code>--thr</code>) and non-maximum suppression (<code>--nms</code>) values. </p>
</div></div><!-- contents -->
<!-- HTML footer for doxygen 1.8.6-->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Fri Apr 2 2021 11:36:34 for OpenCV by &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="../../doxygen.png" alt="doxygen"/>
</a> 1.8.13
</small></address>
<script type="text/javascript">
//<![CDATA[
addTutorialsButtons();
//]]>
</script>
</body>
</html>
